Electronic device and operation method thereof

ABSTRACT

An electronic device worn by a user includes a camera module capable of changing a photographing direction, a motion sensor configured to obtain motion information about the electronic device, an eye-tracking sensor configured to obtain gaze information about eyes of the user, a memory storing one or more instructions, and a processor configured to execute the one or more instructions, wherein the processor is configured to, by executing the one or more instructions, determine the photographing direction based on the gaze information, determine photographing control information for changing the photographing direction of the camera module to obtain an image captured without shaking, based on the motion information, control the camera module to change the photographing, based on the determined photographing control information, and obtain the image captured by the camera module based on the changed photographing direction.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a national stage of International Application No.PCT/KR2021/005627 designating the United States, filed on May 4, 2021,in the Korean Intellectual Property Receiving Office and claimingpriority to Korean Patent Application No. 10-2020-0056155, filed on May11, 2020, in the Korean Intellectual Property Office, the contents ofwhich are incorporated by reference herein in their entireties.

BACKGROUND Field

Various embodiments relate to an electronic device configured to capturean image without shaking by using a camera module embedded in theelectronic device, and an operating method of the electronic device.

Description of Related Art

According to recent technological advances, various types of wearableelectronic devices have been developed. Augmented reality devices enablea user to see both a real scene and a virtual image on a see-throughtype display module placed in front of the eyes of the user wearing theaugmented reality device on his or her face or head. Augmented realityrelates to a technology for projecting a virtual image on a real worldphysical environment space or a real world object to show a singleimage.

A camera module embedded in an augmented reality device may capture areal scene that a user is looking at. Because the real scene is capturedwhile the user is wearing the augmented reality device on his or herhead, there has been a need for research on a method for capturing animage without shaking.

SUMMARY

The disclosure provides an electronic device configured to capture animage without shaking by using a camera module embedded in theelectronic device and an operating method of the electronic device.

The disclosure also provides an electronic device configured to capturean image according to a gaze direction of user's eyes and an operatingmethod of the electronic device.

The technical problems to be solved are not limited to the foregoing,and other unstated technical problems may also exist.

In accordance with an aspect of the disclosure, an electronic deviceworn by a user includes a camera module capable of changing aphotographing direction; a motion sensor configured to obtain motioninformation about the electronic device; an eye-tracking sensorconfigured to obtain gaze information about eyes of the user; a memorystoring one or more instructions; and a processor configured to executethe one or more instructions, wherein the processor is configured to, byexecuting the one or more instructions: determine the photographingdirection based on the gaze information, determine photographing controlinformation for changing the photographing direction of the cameramodule to obtain an image captured without shaking, based on the motioninformation, control the camera module to change the photographingdirection, based on the determined photographing control information,and obtain the image captured by the camera module based on the changedphotographing direction.

The processor may be further configured to, by executing the one or moreinstructions, identify an eye movement state of the user based on theobtained gaze information, and determine the photographing controlinformation based on the identified eye movement state.

The processor may be further configured to, by executing the one or moreinstructions, calculate an eye movement acceleration based on the gazeinformation, and identify the eye movement state based on a result ofcomparing the calculated eye movement acceleration with a predeterminedacceleration threshold value.

The processor may be further configured to, by executing the one or moreinstructions, determine that the eyes of the user are in a state inwhich the eyes are not looking at a particular object when thecalculated eye movement acceleration is greater than the predeterminedacceleration threshold value, and determine the photographing controlinformation based on the motion information and without considering thegaze information.

The processor may be further configured to, by executing the one or moreinstructions, determine that the eyes of the user are in a state inwhich the eyes are looking at a particular object when the calculatedeye movement acceleration is less than or equal to the predeterminedacceleration threshold value, and determine the photographing controlinformation based on the gaze information and the motion information.

The motion information may include information about at least one fromamong a direction, an angle, and a speed of movement of the electronicdevice.

The photographing control information may include information about atleast one from among a direction, an angle, and a speed of rotation ofthe camera module.

The processor may be further configured to, by executing the one or moreinstructions, determine the photographing control information such thatthe photographing direction of the camera module is changed to anopposite direction of a movement direction of the electronic device,based on the motion information.

In accordance with an aspect of the disclosure, an operating method ofan electronic device worn by a user includes determining a photographingdirection of a camera module based on gaze information about eyes of theuser; determining photographing control information for changing thephotographing direction of the camera module to obtain an image capturedwithout shaking, based on motion information about the electronicdevice; controlling the camera module to change the photographingdirection, based on the determined photographing control information;and obtaining the image captured by the camera module based on thechanged photographing direction.

The determining of the photographing control information may includeidentifying an eye movement state of the user based on the obtained gazeinformation; and determining the photographing control information basedon the identified eye movement state.

The identifying of the eye movement state may include calculating an eyemovement acceleration based on the gaze information; and identifying theeye movement state based on a result of comparing the calculated eyemovement acceleration with a predetermined acceleration threshold value.

The identifying of the eye movement state may further includedetermining that the eyes of the user are in a state in which the eyesare not looking at a particular object when the calculated eye movementacceleration is greater than the predetermined acceleration thresholdvalue, and when the eyes of the user are in the state in which the eyesare not looking at the particular object, the determining of thephotographing control information comprises determining thephotographing control information based on the motion information andwithout considering the gaze information.

The identifying of the eye movement state may further includedetermining that the eyes of the user are in a state in which the eyesare looking at a particular object when the calculated eye movementacceleration is less than or equal to the predetermined accelerationthreshold value, and the determining of the photographing controlinformation may include determining the photographing controlinformation based on the gaze information and the motion informationwhen the eyes of the user are in the state in which the eyes are lookingat the particular object.

The determining of the photographing control information may includedetermining the photographing control information such that thephotographing direction of the camera module is changed to an oppositedirection of a movement direction of the electronic device, based on themotion information.

The motion information about the electronic device may comprise at leastone of a direction, an angle or a speed of movement of the electronicdevice.

The photographing control information may comprise at least one of adirection, an angle, or a speed of rotation of the camera module.

A non-transitory computer-readable recording medium may store a programthat is executed by a processor to perform a method comprisingdetermining a photographing direction of a camera module based on gazeinformation about eyes of a user; determining photographing controlinformation for changing the photographing direction of the cameramodule to obtain an image captured without shaking, based on motioninformation about the electronic device worn by the user; controllingthe camera module to change the photographing direction, based on thedetermined photographing control information; and obtaining the imagecaptured by the camera module based on the changed photographingdirection.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram schematically illustrating an embodiment of thedisclosure.

FIG. 2 is a diagram illustrating operations of an electronic deviceaccording to an embodiment.

FIG. 3 is a diagram illustrating an example in which a camera module inan electronic device is rotated, according to an embodiment.

FIG. 4A is a diagram illustrating an example in which gaze informationis obtained according to an embodiment.

FIG. 4B is a diagram illustrating another example in which gazeinformation is obtained according to an embodiment.

FIG. 4C is a diagram illustrating a method of performing calibration ofan eye tracking sensor according to an embodiment.

FIG. 5 is a flowchart illustrating an operation of changing aphotographing direction of a camera module according to an embodiment.

FIG. 6 is a diagram illustrating control of a photographing directionaccording to an embodiment.

FIG. 7 is a flowchart illustrating an operation of controlling aphotographing direction based on an eye movement state according to anembodiment.

FIG. 8A is a diagram illustrating a movement distance according to aneye movement state.

FIG. 8B is a diagram illustrating a movement speed of eyes according toan eye movement state.

FIG. 8C is a diagram illustrating a movement acceleration of eyesaccording to an eye movement state.

FIG. 9 is a flowchart illustrating a method of identifying an eyemovement state according to an embodiment.

FIG. 10A is a diagram illustrating an example of a gaze movement of auser according to an embodiment.

FIG. 10B is a diagram illustrating an example in which an eye movementstate is identified based on eye movement acceleration according to anembodiment.

FIG. 11 is a flowchart illustrating a method of identifying an eyemovement state according to an embodiment.

FIG. 12A is a diagram illustrating an example of a gaze movement of auser according to an embodiment

FIG. 12B is a diagram illustrating an example in which an eye movementstate is identified based on eye movement acceleration according to anembodiment.

FIG. 12C is a diagram illustrating an example in which an eye movementstate is identified based on an eye movement speed according to anembodiment.

FIG. 13 is a block diagram of an electronic device according to anembodiment.

FIG. 14 is a block diagram of an electronic device and an externaldevice according to an embodiment.

DETAILED DESCRIPTION

Hereinafter, embodiments of the disclosure are described in detail withreference to the accompanying drawings so that a person skilled in theart may easily implement the embodiments of the disclosure. However, thedisclosure may be implemented in various different forms and is notlimited to the embodiments described herein. In the drawings, to clearlydescribe the disclosure, any portion irrelevant to the description isomitted, and like reference numerals denote like elements.

General terms which are currently used widely have been selected for usein consideration of their functions in embodiments; however, such termsmay be changed according to an intention of a person skilled in the art,precedents, advent of new technologies, etc. Accordingly, the terms usedin the disclosure should be defined based on their meanings and overalldescriptions of the disclosure, not simply by their names.

Furthermore, while such terms as “first,” “second,” etc., may be used todescribe various components, such components must not be limited to theabove terms. The above terms are used only to distinguish one componentfrom another.

The terms used in the disclosure are used merely to describe aparticular embodiment and not intended to pose a limitation on thedisclosure. As used herein, the singular forms “a,” “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. Throughout the specification, when a portion isdescribed as being connected to another portion, the portion may beconnected directly to another portion, or electrically connected toanother portion with an intervening portion therebetween. Further, whena portion “includes” an element, another element may be furtherincluded, rather than excluding the existence of other elements, unlessotherwise described.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing embodiments (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural. Also, the steps of all methods described herein can be performedin any suitable order unless otherwise indicated herein or otherwiseclearly contradicted by context. The disclosure is not limited to thedescribed order of the steps.

The expressions “according to some embodiments,” “according to anembodiment of the disclosure,” etc. used throughout the specificationare not intended to refer to the same embodiment.

Embodiments may be described in terms of functional block components andvarious processing steps. Such functional blocks may be realized by anynumber of hardware and/or software components configured to perform thespecified functions. For example, the functional blocks of thedisclosure may be implemented by one or more microprocessors or bycircuit components for certain functions. Furthermore, the functionalblocks of the disclosure may be implemented with various programming orscripting languages. The functional blocks may be implemented inalgorithms that are executed on one or more processors. Furthermore,embodiments could employ any number of conventional techniques forelectronics configuration, signal processing and/or control, dataprocessing and the like. The words “mechanism,” “element,” “means,” and“configuration” are used broadly and are not limited to mechanical orphysical embodiments.

Furthermore, the connecting lines, or connectors shown in the variousfigures presented are intended to represent example functionalrelationships and/or physical or logical couplings between the variouselements. It should be noted that many alternative or additionalfunctional relationships, physical connections or logical connectionsmay be present in a practical device.

In the disclosure, the augmented reality (AR) may refer to thetechnology used to show a virtual image in a real world physicalenvironment space or a real world object along with a virtual image.

In addition, the AR device may refer to a device capable of implementingthe AR and may include not only glass-type AR devices, which arewearable on the facial area of a user, such as AR glasses, but also ahead mounted display apparatus or an AR helmet, etc. which are wearableon the head.

The real scene may refer to a real world scene seen by a user throughthe AR device and may include a real world object. Moreover, the virtualimage may refer to an image generated through an optical engine and mayinclude both of a static image and a dynamic image. Such virtual imagemay be observed together with a real scene and may refer to an imagedisplaying information about a real object in the real scene orinformation or control menu, etc. regarding operations of the AR device.

Accordingly, common AR devices may include an optical engine configuredto generate a virtual image constituted by light generated from a lightsource and a waveguide formed of transparent material so that thevirtual image generated by the optical engine is guided to eyes of auser and the user can see the virtual image together with a real worldscene. As described above, the real world scene also needs to beobservable through the AR device. Thus, to guide the light generated bythe optical engine to the user's eyes through the waveguide, an opticalelement configured to change an optical path of light whichfundamentally has straightness may be required. The optical path may bechanged through reflection by a mirror, etc. or through diffraction by adiffractive optical element (DOE), a holographic optical element (HOE),etc.; however, the disclosure is not limited thereto.

Moreover, according to an embodiment of the disclosure, gaze informationmay be information obtained by an eye-tracking sensor 160 (FIG. 2 ) andmay include at least one of a gaze direction of eyes of the user, aposition of user's pupil, and coordinates of a center point of thepupil.

According to an embodiment of the disclosure, motion information may beinformation obtained by a motion sensor 150 of the electronic device 100and may include at least one of a direction, an angle, a speed, and anacceleration of movement of the electronic device 100.

According to an embodiment of the disclosure, photographing controlinformation may be information for changing the photographing directionof the camera module 175 and may include information regarding at leastone of a rotation direction, a rotation angle, and a rotation speed ofrotation of the camera module 175.

The disclosure will now be described more fully with reference to theaccompanying drawings.

FIG. 1 is a diagram schematically illustrating operations of anelectronic device according to an embodiment of the disclosure.

The electronic device 100 according to an embodiment of the disclosuremay be an augmented reality device. For example, as illustrated in FIG.1 , the electronic device 100 may be a device implemented in the form ofglasses which are wearable on the facial area of a user. Moreover, theelectronic device 100 may be a device implemented in the form of agoggle, a helmet, a hat, etc., which are wearable on the head of theuser. However, the disclosure is not limited thereto.

According to an embodiment of the disclosure, the electronic device 100worn by the user may obtain an image of the periphery of the electronicdevice 100 by driving the camera module 175.

When the user wearing the electronic device 100 in his or her daily lifeintends to capture an image of the periphery by using the camera module175 mounted on the electronic device 100, it may be not easy for theelectronic device 100 worn on the facial area of the user to remainsteady without any shaking. When the electronic device 100 is worn onthe facial area or head of the user, the electronic device 100 may movealong with the movement of the head of the user. When the camera module175 is fixed to the electronic device 100, the camera module 175 maymove along with the electronic device 100 and an image may be capturedat the time when the camera module 175 is shaking. In such a case, animage with degraded image quality or an unfocused image may be obtained.

According to an embodiment of the disclosure, the camera module 175 ofthe electronic device 100 may be mounted on a partial area of theelectronic device 100 in a structure rotatable not only vertically orhorizontally but also in a direction of a certain angle so that aphotographing direction of the camera module 175 may be changed. Asillustrated in FIG. 1 , the camera module 175 may be embedded in an areabetween a left eye lens portion and a right eye lens portion in a framearea of the electronic device 100 implemented in the form of glasses,however, the disclosure is not limited thereto.

According to an embodiment of the disclosure, when the electronic device100 senses a movement, the photographing direction may be changed byrotating the camera module 175 in an opposite direction of a movementdirection of the electronic device 100 to compensate for the movement ofthe electronic device 100. In this manner, an image with minimum shakingmay be obtained.

With reference to FIG. 1 , when the gaze of the user wearing theelectronic device 100 on his or her facial area is directed to the front(in a +Z direction), the camera module 175 may capture an image of theperiphery of the electronic device 100 in a photographable angle rangein the front direction (+Z direction) of the electronic device 100according to the gaze direction of the user.

According to an embodiment of the disclosure, when the electronic device100 senses a movement of the electronic device 100, the electronicdevice 100 may rotate the camera module 175 in a direction opposite tothe movement of the electronic device 100 to change the photographingdirection. For example, when the user lowers his or her head whilegazing in the front direction, the electronic device 100 may sense achange of pitch of the electronic device 100 around an X axis in a −Ydirection. At this time, the electronic device 100 may rotate the cameramodule 175 in the direction opposite to the movement of the electronicdevice 100 by controlling the camera module 175 to rotate around the Xaxis in a +Y direction. Accordingly, the electronic device 100 mayobtain an image in focus without degradation of quality.

As illustrated in FIG. 1 , when the electronic device 100 is worn by theuser, the X axis may be a reference axis horizontally crossing theelectronic device 100, the Y axis may be a reference axis verticallycrossing the electronic device 100, and the Z axis may be reference axiscrossing the electronic device 100 from the front to the rear on athree-dimensional (3D) space. The X axis, the Y axis and the Z axis maybe perpendicular to each other.

For example, when the user nods his or her head, such movement may bedescribed as a pitch around the X axis. Moreover, for example, when theuser shakes his or her face left to right, this may be described as ayaw around the Y axis. When the user tilts his or her head to eithershoulder, this may be described as a roll around the Z axis.

Furthermore, according to an embodiment of the disclosure, the cameramodule 175 mounted to a front portion of the electronic device 100 mayperform the pitch around the X axis, the yaw around the Y axis, and theroll around the Z axis.

FIG. 2 is a diagram illustrating operations of an electronic deviceaccording to an embodiment of the disclosure.

With reference to FIG. 2 , the electronic device 100 according to anembodiment of the disclosure may include the camera module 175, aneye-tracking sensor 160, a motion sensor 150, a memory 130, and aprocessor 120. The processor 120 of the electronic device 100 mayexecute programs stored in the memory 130 to control the camera module175, the electronic device 100, the motion sensor 150, etc. in general.

The camera module 175 according to an embodiment of the disclosure maycapture an image of the periphery of the electronic device 100. When anapplication requiring the capturing function is executed, the cameramodule 175 may obtain an image frame of a static image or a video, etc.through an image sensor. An image captured by the image sensor may beprocessed by the processor 120 or a separate image processor.

The camera module 175 according to an embodiment of the disclosure maychange the photographing direction. The camera module 175 may include ahardware structure capable of changing the photographing direction bypanning or tilting the structure. The camera module 175 may rotate notonly vertically or horizontally but also in the clockwise orcounterclockwise direction around a certain axis.

According to an embodiment of the disclosure, by rotating the cameramodule 175 by a certain angle in a certain direction through the controlby the processor 120, the photographing direction may be changed and animage of the periphery may be captured. Moreover, the camera module 175may be rotated in a certain direction by the control by the processor120 and sequentially capture images of the periphery of the electronicdevice 100.

The eye-tracking sensor 160 according to an embodiment of the disclosuremay detect gaze information of the eyes of the user. According to anembodiment of the disclosure, the gaze information may include at leastone from among a gaze direction of the user's eyes, a position of pupilsof the user's eyes, and coordinates of the centers of the pupils.

The eye-tracking sensor 160 may provide light to the eye of the user(the left eye or the right eye) and sense the quantity of lightreflected from the eye of the user. The eye-tracking sensor 160 maydetect a gaze direction of the user's eyes, a position of pupils of theuser's eyes, coordinates of the centers of the pupils, etc. based on thesensed quantity of light.

The eye-tracking sensor 160 may provide light to the user's eyes andcapture an image of the user's eyes. The eye-tracking sensor 160 maydetect a gaze direction of the user's eyes, a position of pupils of theuser's eyes, coordinates of the centers of the pupils, etc. based on thecaptured image of user's eyes.

According to an embodiment of the disclosure, the eye-tracking sensor160 may obtain gaze information of the user by sensing the eyes of theuser wearing the eye-tracking sensor 160 at a predetermined timeinterval.

The motion sensor 150 according to an embodiment of the disclosure mayobtain motion information by sensing movements of the electronic device100. The motion information may include at least one from among adirection, angle, speed, and acceleration of the movement of theelectronic device 100.

The motion sensor 150 may be an inertial measurement unit (IMU). The IMUmay be a combination of sensors configured to sense movements of anobject in a 3D space, e.g., changes in position and orientation. Forexample, the combination of sensors may include an accelerometer, anangular speedometer, a magnetometer, and a gyroscope.

The motion sensor 150 may include at least one of an accelerationsensor, a magnetic sensor, and a gyroscope sensor.

According to an embodiment of the disclosure, the motion sensor 150 mayobtain the motion information by sensing the movement of the electronicdevice 100 at a predetermined time interval.

The memory 130 according to an embodiment of the disclosure may store aprogram to be executed by the processor 120 and store data input to theelectronic device 100 or output from the electronic device 100.

The memory may include at least one storage medium from a flash memorytype storage medium, hard disk type storage medium, multimedia cardmicro type storage medium, card type memory (e.g., SD or XD memory,etc., random access memory (RAM), static RAM (SRAM), read-only m (ROM),electrically erasable programmable ROM (EEPROM), programmable ROM(PROM), magnetic memory, magnetic disk, and optical disk.

The programs stored in the memory 130 may be classified into a pluralityof software modules according to their function, e.g., an eye movementidentification module 131, a photographing direction determinationmodule 132, and a photographing module 133; however, the disclosure isnot limited thereto, and the memory 130 may store some of the foregoingmodules or further include other software modules.

Moreover, according to an embodiment of the disclosure, when the usergazes at each of points included in virtual images (e.g., VI1, VI2, andVI3 in FIG. 4C) in the process of eye-tracking calibration of FIG. 4C,the memory 130 may store information output from an eye-tracking sensor161 (gaze information) in a form of a table as reference gazeinformation. The processor 120 may compare the reference gazeinformation obtained in the process of eye-tracking calibrationprestored in the memory 130 with the gaze information obtained from theeye-tracking sensor 160 to obtain a position or coordinates (2Dcoordinates x, y) of the gaze of the user.

Moreover, the memory 130 may store a reference acceleration thresholdvalue. The processor 120 may use a preset acceleration threshold valuestored in the memory 130 to identify whether the user is looking at aparticular object.

Furthermore, the memory 130 may store a speed threshold value, which isa reference value for identifying a state of the eyes (e.g., a fixationstate, a pursuit state, etc.) The processor 120 may use the preset speedthreshold value stored in the memory 130 to identify whether the eyemovement state of the user is the fixation state or the pursuit state.

The processor 120 may control all operations of the electronic device100. For example, the processor 120 may execute instructions or programsstored in the memory 130 to control the camera module 175, theelectronic device 100, the motion sensor 150, etc. in general.

The processor 120 may include at least one hardware from a centralprocessing unit (CPU), a microprocessor, a graphic processing unit(GPU), application specific integrated circuits (ASICs), digital signalprocessors (DSPs), digital signal processing devices (DSPDs),programmable logic devices (PLDs), and field programmable gate arrays(FPGAs); however, the disclosure is not limited thereto.

The processor 120 may identify the eye movement state of the user byexecuting the eye movement identification module 131 stored in thememory 130.

The processor 120 may compare the reference gaze information prestoredin the memory 130 with the gaze information obtained from theeye-tracking sensor 160 to obtain the position or coordinates (2Dcoordinates (x, y)) of the gaze of the user. The processor 120 maycalculate the speed and acceleration of the eye movement by using theobtained position or coordinates of the gaze. Moreover, the processor120 may identify the eye movement state of the user based on a result ofcomparing the speed and acceleration of the eye movement with each ofthe predetermined speed threshold value and the predeterminedacceleration threshold value.

According to an embodiment of the disclosure, the processor 120 maydetermine that the user's eyes are not looking at a particular objectwhen the eye movement acceleration is greater than the predeterminedacceleration threshold value.

According to an embodiment of the disclosure, the state in which theuser's eyes are not looking at a particular object may include a saccadestate of eyes. The saccade state may refer to a quick movement of user'sgaze from one fixed position to another fixed position in an instant.

Furthermore, the processor 120 may determine that the user's eyes arelooking at a particular object when the eye movement acceleration isless than or equal to the acceleration threshold value.

According to an embodiment of the disclosure, the state in which theuser's eyes are looking at a particular object may be a fixation stateor a pursuit state of the eyes. The fixation state may refer to a statein which pupils stay at a particular point during a certain time periodwhen the gaze of the user is fixed on an object and minute movements ofthe eyes are occurred. The pursuit state may refer to a state of eyes ofthe user when the gaze follows the movement of an object at which theuser's eyes are staring.

According to an embodiment of the disclosure, the processor 120 maydetermine the eye movement state to be the fixation state when the eyemovement acceleration is less than or equal to the accelerationthreshold value and the eye movement speed is less than or equal to thespeed threshold value. Moreover, the processor 120 may determine the eyemovement state to be the pursuit state when the eye movementacceleration is less than or equal to the acceleration threshold valueand the eye movement speed is greater than the speed threshold value.

The processor 120 may determine the photographing direction of thecamera module 175 by executing the photographing direction determinationmodule 132 stored in the memory 130.

According to an embodiment of the disclosure, the processor 120 maydetermine the photographing direction of the camera module 175 to bedirected in the gaze direction of the user based on the gaze informationwhen the operation of the camera module 175 is started. Alternatively,when the operation of the camera module 175 is initiated, the processor120 may determine the photographing direction of the camera module 175based on a default photographing direction.

Furthermore, by executing the photographing direction determinationmodule 132, the processor 120 may determine the photographing controlinformation for changing the photographing direction of the cameramodule 175 to obtain an image captured without shaking. Thephotographing control information may include information about at leastone from among a direction, an angle, and a speed of rotation of thecamera module 175.

According to an embodiment of the disclosure, the processor 120 maydetermine the photographing control information based on the motioninformation of the electronic device 100 so that the photographingdirection of the camera module 175 is changed to a direction opposite tothe movement direction of the electronic device 100.

Moreover, according to an embodiment of the disclosure, the processor120 may determine the photographing control information based on the eyemovement state of the user by executing the photographing directiondetermination module 132.

When it is determined by the eye movement identification module 131 thatthe eyes of the user are looking at a particular object, the processor120 may determine the photographing control information based on thegaze information and the motion information of the electronic device100.

When it is determined by the eye movement identification module 131 thatthe eyes of the user are not looking at a particular object, theprocessor 120 may determine the photographing control information basedon the motion information of the electronic device 100 withoutconsidering the gaze information.

The processor 120 may capture an image of the periphery of theelectronic device 100 through the camera module 175 by executing thephotographing module 133 stored in the memory 130.

The processor 120 may capture an image of the periphery of theelectronic device 100 based on the photographing direction determined bythe photographing direction determination module 132.

According to an embodiment of the disclosure, the processor 120 maychange the photographing direction of the camera module 175 based on thephotographing control information to capture an image of the peripheryof the electronic device 100. Moreover, the processor 120 may capture animage of the periphery of the electronic device 100 while changing thephotographing direction of the camera module 175 by a certain anglebased on the photographing control information.

FIG. 3 is a diagram illustrating an example in which a camera module inan electronic device is rotated, according to an embodiment of thedisclosure.

FIG. 3 is a diagram schematically illustrating a structure of an upperportion 175 a and a lateral portion 175 b of the camera module 175embedded in the electronic device 100; however, the disclosure is notlimited thereto.

With reference to FIG. 3 , the camera module 175 may be mounted onto theelectronic device 100 and have a structure in which the camera module175 may be panned or tilted by a certain angle in a certain directionwith respect to the a rotation center 31. Accordingly, the camera module175 may be tilted in a certain direction around the rotation center 31by a certain angle or may rotate in the clockwise or counterclockwisedirection.

According to an embodiment of the disclosure, as illustrated in FIG. 1 ,when the camera module 175 is embedded in the front portion of theelectronic device 100, the camera module 175 may rotate around the Xaxis, the Y axis, or the Z axis of the electronic device 100 worn by theuser.

FIG. 3 illustrates an example of the camera module 175; however, thedisclosure is not limited thereto, and the camera module 175 may bemounted onto the electronic device 100 and have a rotatable structure tochange the photographing direction.

FIG. 4A is a diagram illustrating an example in which gaze informationis obtained according to an embodiment of the disclosure.

FIG. 4A illustrates a method of tracking the gaze of the user based onthe quantity of reflected light from the eyes of the user.

According to an embodiment of the disclosure, the electronic device 100may include a first eye-tracking sensor 161 configured to track the gazeof one eye and a second eye-tracking sensor 162 configured to track thegaze of the other eye. As the first eye-tracking sensor 161 and thesecond eye-tracking sensor 162 may have the same structure and operatein the same manner, FIG. 4A is described focusing on the firsteye-tracking sensor 161.

With reference to FIG. 4A, the first eye-tracking sensor 161 accordingto an embodiment of the disclosure may include a lighting portion 301configured to provide light to the user's eyes and a sensing portion 302configured to sense the light. The lighting portion 301 may include alight source providing light and a scanning mirror controlling adirection of light provided from the light source. The scanning mirrormay control the direction of light provided from the light source to bedirected toward the eyes 320 of the user (e.g., pupils 310). Thescanning mirror may include a structure capable of mechanically changinga reflection angle by reflecting the light provided from the lightsource so that the light is directed towards the eyes 320 of the userand may scan an area including the pupils 310 by using the lightprovided from the light source according to the changed reflectionangle.

The sensing portion 302 may sense light reflected from the eyes 320 ofthe user and measure the quantity of the sensed light. For example, whenthe light is reflected from the center of the pupils 310 of the user,the quantity of light sensed by the sensing portion 302 may be themaximum quantity. Accordingly, when the quantity of light sensed by thesensing portion 302 is the maximum quantity, the first eye-trackingsensor 161 may determine a direction 340 of the gaze of the user basedon a point 330 at which the light is incident on and reflected from theuser's eyes. For example, when the quantity of light is the maximumquantity, the first eye-tracking sensor 161 may determine the direction340 connecting the point 330 at which the light is incident on andreflected from the user's eyes with the center point of the eyes 320 ofthe user to be the direction of the gaze of the user's eye (e.g., theleft eye of the user.)

Moreover, the second eye-tracking sensor 162 may also determine thedirection of the gaze of the user's eye (e.g., the right eye) in thesame manner described with reference to FIG. 4A.

According to an embodiment of the disclosure, the processor 120 mayobtain the gaze information of the left or right eye from one of thefirst eye-tracking sensor 161 and the second eye-tracking sensor 162 anduse the obtained gaze information when determining the photographingdirection or the photographing control information.

FIG. 4B is a diagram illustrating another example in which gazeinformation is obtained according to an embodiment of the disclosure.The first eye-tracking sensor 161 according to an embodiment may includea lighting portion 351 and a capturing portion 352.

FIG. 4B illustrates a method of tracking the gaze of the user based onthe position of reflected light from the eyes of the user.

The lighting portion 351 according to an embodiment of the disclosuremay include an infrared light-emitting diode (IR LED), etc. Asillustrated in FIG. 4B, the lighting portion 351 may include a pluralityof LEDs respectively arranged at different positions. The lightingportion 351 may provide light to the user's eyes (e.g., infrared light)when capturing an image of the user's eyes. As light is provided to theuser's eyes, reflected light may be generated from the user's eyes.

Moreover, the capturing portion 352 may include at least one camera andthe at least one camera may include an IR camera. The electronic device100 may use an image of user's eyes captured by the capturing portion352 to track the gaze of the user's eye (e.g., the left eye of theuser). For example, the first eye-tracking sensor 161 may track the gazeof the user by detecting the pupil and the reflected light from theimage of user's eyes. The first eye-tracking sensor 161 may detect theposition of the pupil and the position of the reflected light of theimage of the user's eyes and determine the gaze direction of the user'seyes based on a relation between the position of the pupil and theposition of the reflected light.

For example, the first eye-tracking sensor 161 may detect pupils 370 andreflected light 381 from a captured first eye image 361 and determine agaze direction 391 of eyes of the user based on a relation between theposition of the pupils 370 and the position of the reflected light 381.In the same manner, the pupils 370 and reflected light 382, 383, 384,and 385 may be detected from each of second to fifth eye images 362,363, 364, and 365, and based on a relation between the position of thepupils 370 and the position of the reflected light, gaze directions 392,393, 394, and 395 of the user's eyes may be determined.

Moreover, the second eye-tracking sensor 162 may also determine thedirection of the gaze of the user's eye (e.g., the right eye) in thesame manner described with reference to FIG. 4B.

According to an embodiment of the disclosure, the processor 120 obtainthe gaze information of the left or right eye from one of the firsteye-tracking sensor 161 and the second eye-tracking sensor 162 and mayuse the obtained gaze information when determining the photographingdirection or the photographing control information.

FIG. 4C is a diagram illustrating a method of performing calibration ofan eye tracking sensor according to an embodiment of the disclosure.

When the user uses the electronic device 100 for the first time, toaccurately measure the gaze of the left eye and the right eye of theuser, calibration of the first eye-tracking sensor 161 and the secondeye-tracking sensor 162 may be performed. The electronic device 100 mayoutput virtual images VI1, VI2, and VI3 each having different depths(e.g., d1, d2, and d3) represented by a plurality of points (generally,nine points) to induce the user to gaze at each of the plurality ofpoints.

When the user gazes at each point included in the virtual images VI1,VI2, and VI3, the electronic device 100 may prestore information outputfrom the eye-tracking sensor 161 in the form of a table as referencegaze information.

The processor 120 of the electronic device 100 may compare the referencegaze information prestored in the memory 130 with the gaze informationoutput from the eye-tracking sensor 161 to determine the position orcoordinates (2D coordinates (x, y)) of the gaze of the user. Theprocessor 120 may use the gaze information output from the firsteye-tracking sensor 161 to determine coordinates of the left eye. Theprocessor 120 may use the gaze information output from the secondeye-tracking sensor 162 to determine coordinates of the right eye.

According to an embodiment of the disclosure, the processor 120 of theelectronic device 100 may use the coordinates of the gaze of either eyeto calculate the eye movement speed and acceleration.

FIG. 5 is a flowchart illustrating an operation of changing aphotographing direction of a camera module according to an embodiment ofthe disclosure. FIG. 6 is a diagram illustrating control of aphotographing direction according to an embodiment of the disclosure.FIG. 6 illustrates the method of FIG. 5 .

In operation S501 of FIG. 5 , the electronic device 100 may determinethe photographing direction based on the gaze information.

According to an embodiment of the disclosure, the processor 120 of theelectronic device 100 may obtain the gaze information from theeye-tracking sensor 160 (FIG. 1 ). The gaze information may include atleast one of a gaze direction of the user's eyes, a position of pupils,and coordinates of center points of the pupils.

When the user is wearing the electronic device 100 on his or her facialarea or head, the processor 120 may determine the photographingdirection based on the gaze information obtained from the eye-trackingsensor 160 so that the camera module 175 captures a real scene at whichthe user is gazing. The processor 120 may determine the photographingdirection so that the camera module 175 is directed in the user's gazedirection when the operation of the camera module 175 is started.

In operation S502 of FIG. 5 , the photographing control information forchanging the photographing direction of the camera to obtain an imagecaptured without shaking may be determined based on the motioninformation.

The electronic device 100 may obtain the motion information from themotion sensor 150 (FIG. 1 ). The motion information may include at leastone of a direction, angle, speed, and acceleration of the movement ofthe electronic device 100.

When the motion information of the electronic device 100 is obtainedfrom the motion sensor 150, the processor 120 may determine thephotographing control information so that the camera module 175 rotatesin a direction and by an angle to compensate for the movement of theelectronic device 100, i.e., in a direction opposite to the movementdirection of the electronic device 100 by the same angle the electronicdevice 100 has moved. According to an embodiment of the disclosure, thephotographing control information may include information about at leastone of a direction, an angle, and a speed of rotation of the cameramodule 175.

For example, when the movement of the electronic device 100 around the Yaxis in the −X direction by 5 degrees is detected, the processor 120 maydetermine the photographing control information including a rotationdirection and a rotation angle of the camera module 175 (e.g., rotationaround the Y axis in the +X direction by 5 degrees).

With reference to FIG. 6 , when the electronic device 100 controls therotation of the camera module 175 considering the movement of theelectronic device 100, a control signal may be determined to rotate thecamera module 175 by an inverse value of a movement signal value of theelectronic device 100. For example, when no movement of the electronicdevice 100 is detected in time sections p1 and p3, the camera module 175may not be rotated, and when movement of the electronic device 100 isdetected in time sections p2 and p4, a rotation control signal may bedetermined to be an inverse value of the movement signal value.

In operation S503 of FIG. 5 , the electronic device 100 may control thecamera module to change the photographing direction based on thedetermined photographing control information. The processor 120 maychange the photographing direction by controlling the camera module 175to rotate in a rotation direction and by a rotation angle included inthe determined photographing control information.

In operation S504 of FIG. 5 , the electronic device 100 may obtain animage captured based on the changed photographing direction. Theprocessor 120 may obtain an image (e.g., a static image, video or apanorama image, etc.) by capturing an image of the periphery of theelectronic device 100 through the camera module 175 which is rotated inthe changed photographing direction.

According to an embodiment of the disclosure, when the electronic device100 the user is wearing on his or her facial area or head is shaking,the electronic device 100 may rotate the camera module 175 to captureand obtain an image with minimum shaking.

FIG. 7 is a flowchart illustrating an operation of controlling aphotographing direction based on an eye movement state according to anembodiment of the disclosure.

In operation S701 of FIG. 7 , the electronic device 100 may obtain thegaze information of the user's eyes.

The processor 120 of the electronic device 100 may obtain the gazeinformation including at least one of a gaze direction of the user'seyes, positions of pupils, and coordinates of the center points of thepupils through the eye-tracking sensor 160.

In operation S702 of FIG. 7 , the electronic device 100 may identify theeye movement state based on the gaze information.

The processor 120 may execute the eye movement identification module 131(FIG. 2 ) to identify the eye movement state by using the gazeinformation obtained from the eye-tracking sensor 160.

According to an embodiment of the disclosure, the eye movement state maybe a state in which the user's eyes are looking at a particular objector a state in which the user's eyes are not looking at a particularobject.

For example, the state in which the user's eyes are looking at aparticular object may be the fixation state of the eyes in which theuser's eyes are fixed on an object and only very minute movement occursor the pursuit state of the eyes in which the user's eyes follow anobject when the object is moving. Furthermore, the state in which theeyes are not looking at a particular object may be the saccade state ofthe eyes in which the gaze of the user quickly moves from one fixedposition to another fixed position.

As described with reference to FIG. 4C, the processor 120 may comparethe reference gaze information prestored in the memory 130 with the gazeinformation obtained from the eye-tracking sensor 160 to determine theposition (2D coordinates (x, y)) of the gaze of the user. The processor120 may calculate the eye movement speed and acceleration by using theposition change amount of user's gaze during a certain time period.

The processor 120 may determine whether the eyes are looking at aparticular object (e.g., the fixation state or the pursuit state of theeyes) or not (e.g., the saccade state of the eyes) based on thecalculated eye movement acceleration.

In operation S703 of FIG. 7 the electronic device 100 may determine thephotographing control information based on the eye movement state.

According to an embodiment of the disclosure, when the eyes of the userare determined to be in the state where they are looking at a particularobject, the processor 120 may determine the photographing controlinformation based on both of the gaze information of the user and themotion information of the electronic device.

For example, when the eyes of the user are in the fixation state, theprocessor 120 may adjust the photographing direction of the cameramodule 175 so that the movement of the electronic device 100 is sensedand an image taken without shaking is obtained while the camera module175 is directed towards an object at which the user is gazing.

Furthermore, for example, when the eyes are in the pursuit state wherethe gaze of the user follows the movement of the object, the processor120 may change the photographing direction so that the camera module 175may capture an image of the moving object at which the user is gazingwhile adjusting the photographing direction of the camera module 175 toobtain an image taken without shaking according to a result of sensingthe movement of the electronic device 100.

According to an embodiment of the disclosure, the processor 120 may seta first weight regarding the gaze information of the user and a secondweight regarding the motion information of the movement of theelectronic device when considering both of the gaze information of theuser and the motion information of the electronic device.

For example, the processor 120 may set the first weight to be higherthan the second weight so that the camera module 175 rotates firstly inthe gaze direction of the user. Alternatively, for example, theprocessor 120 may set the second weight to be higher than the firstweight so that the camera module 175 rotates according to the motioninformation of the electronic device rather than the gaze information ofthe user.

In addition, according to an embodiment of the disclosure, when the eyesof the user are determined to be in the state where they are not lookingat a particular object, the processor 120 may determine thephotographing control information based on the motion information of theelectronic device.

For example, when the eyes of the user are in the saccade state wherethey are not looking at a particular object, the processor 120 maydetermine the photographing control information of the camera module 175to obtain an image taken without shaking according to a result ofsensing the movement of the electronic device 100.

For example, when the user averts his or her eyes to gaze at a secondobject from a first object, in a time interval of the gaze movement fromthe first object to the second object, the processor 120 may identifythe saccade state in which the gaze of the user moves quickly. In thisoccasion, the processor 120 may adjust the photographing direction ofthe camera module 175 to obtain an image taken without shaking accordingto a result of sensing the movement of the electronic device 100 withoutconsidering the movement of the user's gaze.

FIG. 8A is a diagram illustrating a movement distance according to aneye movement state. FIG. 8B is a diagram illustrating a movement speedof eyes according to an eye movement state. FIG. 8C is a diagramillustrating a movement acceleration of eyes according to an eyemovement state.

FIGS. 8A to 8C are graphs showing an eye movement distance, a movementspeed, and a movement acceleration in a time section t1, time sectiont2, and time section t3.

The movement distance from the reference point in FIG. 8A may becalculated by a distance from a certain reference point to thecoordinates of the center point of the pupil. The eye movement speed inFIG. 8B may be calculated by an eye position change amount during acertain time period. The eye movement acceleration in FIG. 8C may becalculated by an eye movement speed change amount during a certain timeperiod.

When the eye movement state is the fixation state (time section t1), thedistance may be fixed within a certain range (FIG. 8A), the movementspeed may be close to 0°/ms (FIG. 8B), and the movement acceleration mayalso be close to 0°/ms² (FIG. 8C).

When the eye movement state is the saccade state (time section t2), thedistance change amount may be beyond a certain range during the t2 timesection (FIG. 8A) and the movement speed and acceleration may also becalculated to be variable values beyond a certain range (FIGS. 8B and8C).

Furthermore, when the eye movement state is the pursuit state (timesection t3), the distance may vary consistently during the time sectiont3, the movement speed may be close to uniform velocity (FIG. 8B), andthe movement acceleration may be close to 0°/ms² (FIG. 8C).

According to an embodiment of the disclosure, the eye movementidentification module 131 stored in the memory 130 may include analgorithm capable of identifying the eye movement state based on the eyemovement speed and the movement acceleration. The processor 120 of theelectronic device 100 may execute the eye movement identification module131 to identify the eye movement state based on calculation results ofthe eye movement speed and movement acceleration by using the gazeinformation obtained through the eye-tracking sensor 160.

FIG. 9 is a flowchart illustrating a method of identifying an eyemovement state according to an embodiment of the disclosure.

The processor 120 of the electronic device 100 may obtain the gazeinformation including a gaze direction of the user's eyes, position ofpupils, and coordinates of the center points of the pupils through theeye-tracking sensor 160. The processor 120 may obtain the gazeinformation through the eye-tracking sensor 160 at preset timeintervals.

Moreover, according to an embodiment of the disclosure, the processor120 of the electronic device 100 may obtain the motion informationthrough the motion sensor 150 (FIG. 1 ) when the movement of theelectronic device 100 is detected. The motion information may include atleast one of a direction, angle, speed, and acceleration of the movementof the electronic device 100.

In operation S901 of FIG. 9 , the electronic device 100 may determinethe photographing control information based on the motion information ofthe electronic device.

According to an embodiment of the disclosure, when the motioninformation of the electronic device 100 is obtained from the motionsensor 150, the processor 120 may determine the photographing controlinformation so that the camera module 175 is rotated in a direction andby an angle to compensate for the movement of the electronic device 100based on the motion information.

In operation S902 of FIG. 9 , the electronic device 100 may calculatethe eye movement acceleration based on the gaze information.

According to an embodiment of the disclosure, the electronic device 100may calculate the eye movement acceleration based on the gazeinformation obtained through the eye-tracking sensor 160.

The processor 120 of the electronic device 100 may calculate the eyemovement speed by calculating the position change amount of pupils overa certain time period based on the position of the pupils of the eyesobtained through the eye-tracking sensor 160. Moreover, the processor120 may calculate the movement acceleration by calculating the movementspeed change amount of the pupils over a certain time period.

In operation S903 of FIG. 9 , the electronic device 100 may identifywhether the eye movement acceleration is less than or equal to apredetermined acceleration threshold value.

According to an embodiment of the disclosure, the memory 130 of theelectronic device 100 may prestore the acceleration threshold valuewhich is a reference value for identifying whether the eyes of the userare looking at a particular object.

The processor 120 of the electronic device 100 may compare the movementacceleration calculated based on the gaze information obtained inoperation S901 with the predetermined acceleration threshold valuestored in the memory 130.

In operation S904 of FIG. 9 , when the eye movement acceleration is lessthan or equal to the predetermined acceleration threshold value, theelectronic device 100 may determine that the user's eyes are looking ata particular object.

When the eye movement acceleration calculated in operation S902 is lessthan or equal to the prestored acceleration threshold value, theprocessor 120 may determine that the eyes are looking at a particularobject (i.e., in the fixation state or in the pursuit state).

With reference to FIGS. 8A to 8C, when the user's eyes are looking at afixed object, i.e., in the fixation state, the eye movement may be in anearly stationary state. In addition, when the user's eyes are lookingat a moving object, i.e., in the pursuit state, the eye movement may bein a nearly uniform state of motion. Accordingly, when the eyes are inthe fixation state or in the pursuit state, the eye movementacceleration may be calculated close to 0°/ms² and less than or equal toa preset acceleration threshold value.

In operation S905 of FIG. 9 , when the eyes of the user are looking at aparticular object, the electronic device 100 may determine thephotographing control information based on the gaze information and themotion information of the electronic device.

According to an embodiment of the disclosure, when the eyes of the userare determined to be in the state where they are looking at a particularobject, the processor 120 may determine the photographing controlinformation so that the photographing direction is changed to adirection in which the user is looking at the particular object.Moreover, when the camera module 175 is directed in a gaze direction ofthe user, the processor 120 may determine the photographing controlinformation to capture an image without shaking according to a result ofsensing the movement of the electronic device 100.

The processor 120 may calculate the eye movement acceleration based onthe gaze information obtained through the eye-tracking sensor 160 atpreset time intervals (S902). The processor 120 may redetermine, at thepreset time intervals, the photographing control information accordingto the eye movement state identified by calculating the eye movementacceleration.

In operation S906 of FIG. 9 , when the eye movement accelerationcalculated in operation S902 is greater than the predeterminedacceleration threshold value, the electronic device 100 may determinethat the user's eyes are not looking at a particular object.

With reference to FIG. 8C, when the eyes are in the saccade state wherethe gaze of the user moves quickly from one fixed position to anotherfixed position, the eye movement may be expressed as a motion withvariable acceleration. Accordingly, when the calculated eye movementacceleration is greater than the preset acceleration threshold value,the eye movement state may be determined to be the saccade state.

When the eyes of the user are determined to be in the state where theyare not looking at a particular object, the processor 120 of theelectronic device 100 may determine the photographing controlinformation based on the motion information of the electronic device(S901).

According to an embodiment of the disclosure, with respect to the timesection during which the user's eyes are in the saccade state where theyare not looking at a particular object, the processor 120 may determinethe photographing control information to obtain an image taken withoutshaking according to a result of sensing the movement of the electronicdevice 100 without considering the gaze movement of the user.

FIG. 10A is a diagram illustrating an example of a gaze movement of auser according to an embodiment of the disclosure. FIG. 10B is a diagramillustrating an example in which an eye movement state is identifiedbased on eye movement acceleration according to an embodiment of thedisclosure.

FIG. 10A illustrates a case where the user gazes at a first object 1001first and then after a certain time period, gazes at a second object1002. FIG. 10B is a graph showing an acceleration value calculated inthe case of FIG. 10A.

The processor 120 of the electronic device 100 may identify, aftercalculating the eye movement acceleration by using the gaze informationobtained through the eye-tracking sensor 160, that the acceleration inthe time section t1 is close to 0°/ms², i.e., less than or equal to thepreset acceleration threshold value. Accordingly, the processor 120 maydetermine that the user's eyes are looking at a particular object in thetime section t1.

Furthermore, the processor 120 may identify, after calculating the eyemovement acceleration by using the gaze information obtained through theeye-tracking sensor 160 in the time section t2, that the movementacceleration is greater than the preset acceleration threshold value.Accordingly, the processor 120 may determine that the user's eyes arenot looking at a particular object in the time section t2.

Furthermore, the processor 120 may identify, after calculating the eyemovement acceleration by using the gaze information obtained through theeye-tracking sensor 160 in the time section t3, that the movementacceleration is close to 0°/ms² i.e., less than the preset accelerationthreshold value. Accordingly, the processor 120 may determine that theuser's eyes are looking at a particular object in the time section t3.

The processor 120 may determine the photographing control information toadjust the photographing direction based on the motion information ofthe electronic device 100 in the time section t1 and the time section t3while the camera module 175 is directed in the gaze direction of theuser.

Moreover, the processor 120 may determine the photographing controlinformation based on the motion information of the electronic device 100in the time section t2 without considering the gaze direction of theuser.

FIG. 11 is a flowchart illustrating a method of identifying an eyemovement state according to an embodiment of the disclosure.

As the electronic device 100 identifies that the eye movementacceleration is less than or equal to the predetermined accelerationthreshold value in operation S1101 of FIG. 11 , the electronic device100 may identify that the eye movement speed is less than or equal tothe predetermined speed threshold value in operation S1102.

With reference to FIGS. 8A to 8C, in a section where the eye movementacceleration is close to 0°/ms², i.e., less than or equal to thepredetermined acceleration threshold value, the user's eyes may be inthe fixation state or in the pursuit state. In both the fixation stateand the pursuit state, the eyes are determined to be looking at aparticular object.

The processor 120 may identify whether the user's eyes are in thefixation state or in the pursuit state by comparing the movement speedcalculated by using the gaze information obtained from the eye-trackingsensor 160 with the speed threshold value, which is a reference speedfor identifying the stationary state or the pursuit state of the eyesprestored in the memory 130.

In operation S1103 of FIG. 11 , when the eye movement speed is less thanor equal to the predetermined speed threshold value, the electronicdevice 100 may determine the eye movement state to be the fixationstate. In operation S1104, when the eye movement speed is greater thanthe predetermined speed threshold value, the electronic device 100 maydetermine the eye movement state to be the pursuit state.

With reference to FIG. 8B, when the eyes are in the fixation state, theeye movement speed may be close to 0°/ms² and less than or equal to thepredetermined speed threshold value. Moreover, when the eyes are in thepursuit state, the eye movement speed may be greater than thepredetermined speed threshold value.

According to an embodiment of the disclosure, when the eyes are in thefixation state or in the pursuit state, the processor 120 may determinethe photographing control information so that the camera module 175 mayadjust the photographing direction based on the motion information ofthe electronic device 100 while tracking the gaze direction of the user.

FIG. 12A is a diagram illustrating an example of a gaze movement of auser according to an embodiment of the disclosure. FIG. 12B is a diagramillustrating an example in which an eye movement state is identifiedbased on eye movement acceleration according to an embodiment of thedisclosure. FIG. 12C is a diagram illustrating an example in which aneye movement state is identified based on an eye movement speedaccording to an embodiment of the disclosure.

FIG. 12A illustrates a case in which when the user is looking at a thirdobject 1201, and the third object 1201 moves in the right direction, thegaze of the user moves along with the moving third object 1201.

FIGS. 12B and 12C are graphs respectively showing an acceleration andspeed value calculated in the case of FIG. 12A.

The processor 120 of the electronic device 100 may identify, aftercalculating the eye movement acceleration by using the gaze informationobtained through the eye-tracking sensor 160, that the acceleration inthe time section t1 and the time section t2 is close to 0°/ms², i.e.,less than or equal to the preset acceleration threshold value.

Moreover, the processor 120 may identify, after calculating the eyemovement speed by using the gaze information obtained through theeye-tracking sensor 160, that the speed is close to 0°/ms in the timesection t1, i.e., less than or equal to the predetermine speed thresholdvalue, and that the movement speed is greater than the predeterminedspeed threshold value in the time section t2. Accordingly, the processor120 may identify that the eyes are in the fixation state in the timesection t1 and in the pursuit state in the time section t2.

The processor 120 may adjust the photographing direction of the cameramodule 175 to obtain an image taken without shaking while sensing themovement of the electronic device 100 in the time section t1 when thecamera module 175 is directed to the third object 1201 at which the useris gazing.

Furthermore, in the time section t2 in which the eyes are in the pursuitstate, the processor 120 may change the photographing directionaccording to the moving gaze direction of the user and simultaneouslyadjust the photographing direction of the camera module 175 to obtain animage taken without shaking while sensing the movement of the electronicdevice 100.

FIG. 13 is a block diagram of an electronic device according to anembodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may be an AR device having a communication function and data processingfunction and providing an AR image, but the disclosure is not limitedthereto.

As illustrated in FIG. 13 , the electronic device 100 according to anembodiment of the disclosure may include the eye-tracking sensor 160,the motion sensor 150, the camera module 175, the memory 130, theprocessor 120, a display 140, a communication portion 180, a microphone190, and a user input portion 195.

However, the components illustrated in FIG. 13 are not essentialcomponents of the electronic device 100. The electronic device 100 maybe implemented by more or fewer components than the componentsillustrated in FIG. 13 .

As the operations of the eye-tracking sensor 160, the motion sensor 150,the camera module 175, the memory 130, and the processor 120 are alreadydescribed with reference to FIG. 2 , any redundant description thereofis omitted.

The display 140 according to an embodiment of the disclosure may outputinformation processed by the processor 120. For example, the display 140may display a virtual object.

According to an embodiment of the disclosure, the display 140 mayprovide an AR image. The display 140 according to an embodiment of thedisclosure may include a waveguide and a display module. The waveguidemay be made of a transparent material showing a partial area of a rearsurface of a wearable device 1 worn by the user. The waveguide mayinclude a single or multi-layer structure plate formed of a transparentmaterial in which light may be reflected and propagated. The waveguidemay face an exit surface of a display module and may receive light of aprojected virtual image. The transparent material may refer to amaterial capable of transmitting light, may not have 100% transparency,and may have a certain color.

According to an embodiment of the disclosure, as the waveguide is formedof a transparent material, the user may not only see a virtual object ofa virtual image through the display 140 but also an actual outsidescene, and thus the waveguide may be referred to as a see-throughdisplay. The display 140 may provide an AR image by outputting a virtualobject of a virtual image through the waveguide.

The communication portion 180 may include one or more componentsconfigured to facilitate communication between the electronic device 100and an external device 200 (FIG. 14 ) or a server.

For example, the communication portion 180 may include a near fieldcommunication portion and a mobile communication portion.

The near field communication portion may include a bluetooth portion, anear field communication portion (NFC/RFID portion), a WLAN (WiFi)communication portion, a zigbee communication portion, an infrared dataassociation (IrDA) communication portion, a ultra wideband (UWB)communication portion, an ant+ communication portion, etc.; however, thedisclosure is not limited thereto.

The mobile communication portion may receive and transmit a wirelesssignal from and to at least one of a base station, an external terminal,and a server on a mobile communication network. The wireless signal mayinclude a voice call signal, a video call signal, or data in variousforms according to receipt and transmission of text/multimedia message.

The electronic device 100 according to an embodiment of the disclosuremay transmit the gaze information and the motion information to theexternal device 200 (FIG. 14 ) through the communication portion 180.Accordingly, the external device 200 may determine the photographingcontrol information based on the gaze information and the motioninformation.

Moreover, according to an embodiment of the disclosure, the electronicdevice 100 may receive the photographing control information determinedat the external device 200 and control the photographing direction ofthe camera module 175 based on the received photographing controlinformation.

According to the embodiment, the electronic device 100 may transmit animage obtained by using the camera module 175 to the external device200. The external device 200 may store the received image.

The microphone 190 may receive an external sound signal and process itinto electrical audio data. For example, the microphone 190 may receivea sound signal from an external device or a speaker. The microphone 190may use various noise removing algorithms to remove noise generated inthe process of receiving the external sound signal. The microphone 190may receive a voice input of the user to control the electronic device100.

The user input portion 195 may be an input tool to which the user inputsdata to control the electronic device 100. For example, the user inputportion 195 may include at least one of a key pad, a dome switch, atouch pad (capacitive type, resistive type, infrared-sensitive type,surface ultrasonic conductive type, integral tension measurement type,piezo effect type, etc.), a jog wheel, and a jog switch, etc.; however,the disclosure is not limited thereto.

The user input portion 195 may receive a user input for capturing animage of the periphery of the electronic device 100 by using the cameramodule 175. Moreover, the user input portion 195 may receive a userinput to receive a service from the electronic device 100 or a serverbased on the captured image.

FIG. 14 is a block diagram of an electronic device and an externaldevice according to an embodiment of the disclosure.

According to an embodiment of the disclosure, the electronic device 100may operate in association with the external device 200. The electronicdevice 100 may transmit the gaze information and the motion informationto the external device 200, and the external device 200 may determinethe photographing control information by using the gaze information andthe motion information, and provide the photographing controlinformation to the electronic device 100.

The components of the electronic device 100 illustrated in FIG. 14 maycorrespond to the components of the electronic device 100 illustrated inFIG. 13 , and thus, description thereon is omitted.

The external device 200 illustrated in FIG. 14 may include a processor220, a memory 230, and a communication portion 280. However, thecomponents illustrated in FIG. 14 are not essential components of theexternal device 200. The external device 200 may be implemented by moreor fewer components than the components illustrated in FIG. 14 .

The processor 220 according to an embodiment of the disclosure maycontrol the external device 200 in general. The processor 220 accordingto an embodiment of the disclosure may execute one or more programsstored in the memory 230.

The memory 230 according to an embodiment of the disclosure may storedata, programs, and applications for driving and controlling theexternal device 200. The programs stored in the memory 230 may includeone or more instructions. The programs (one or more instructions) orapplications stored in the memory 230 may be executed by the processor220.

The memory 230 according to an embodiment of the disclosure may storeprograms for processing and control by the processor 220. The programsstored in the memory 230 may be classified into a plurality of modulesaccording to their function. For example, a software module may bestored in the memory 230 of the external device 200 so that theoperations performed by the eye movement identification module 131 andthe photographing direction determination module 132 of the electronicdevice 100 may be performed by the processor 220 of the external device200.

In addition, the memory 230 may store the gaze information and themotion information received from the electronic device 100. Moreover,the memory 230 may store an image obtained by the camera module 175 andreceived from the electronic device 100.

The communication portion 280 may include one or more componentsconfigured to facilitate communication between the external device 200and the electronic device 100 or between the external device 200 and aserver.

For example, the communication portion 280 may include a near fieldcommunication portion and a mobile communication portion.

The near field communication portion may include a bluetooth portion, anear field communication portion (NFC/RFID portion), a WLAN (WiFi)communication portion, a zigbee communication portion, an infrared dataassociation (IrDA) communication portion, a ultra wideband (UWB)communication portion, an ant+ communication portion, etc.; however, thedisclosure is not limited thereto.

The mobile communication portion may receive and transmit a wirelesssignal from and to at least one of a base station, an external terminal,and a server on a mobile communication network. The wireless signal mayinclude a voice call signal, a video call signal, or data in variousforms according to receipt and transmission of text/multimedia message.

The external device 200 according to an embodiment of the disclosure mayreceive the gaze information and the motion information from theelectronic device 100 through the communication portion 280.Accordingly, the processor 220 of the external device 200 may determinethe photographing control information based on the gaze information andthe motion information. Moreover, the external device 200 may transmitthe determined photographing control information to the electronicdevice 100 through the communication portion 280.

The external device 200 according to an embodiment of the disclosure mayreceive an image captured from the electronic device 100 through thecommunication portion 280. Accordingly, the external device 200 maystore the received image in the memory 230.

The embodiments can be written as computer programs and can beimplemented in general-use digital computers that execute the programsusing a computer-readable recording medium. Moreover, the data structureused in the embodiments may be recorded to the computer-readablerecording medium through various tools. The embodiment may also beimplemented in the form of a computer-readable recording mediumincluding computer-executable commands, such as a computer module. Whensoftware modules or algorithms are involved, these software modules maybe stored as program instructions or computer-readable codes executableon a processor on a computer-readable recording medium.

The computer-readable recording medium may be any recording medium whichcan be accessed by a computer, and include volatile and non-volatilemedia, and separable and non-separable media. Examples of thecomputer-readable recording medium may include magnetic storage media(e.g., ROM, RAM, floppy disks, hard disks, etc.), and optical recordingmedia (e.g., CD-ROMs, digital versatile disks (DVDs), etc.); however,the disclosure is not limited thereto. Further, the computer-readablerecording media may include both of computer storage media andcommunication media.

The computer-readable recording media can also be distributed overnetwork-coupled computer systems so that data, for example, programinstructions and codes stored in the distributed recording media areexecuted by at least one computer.

The particular implementations shown and described herein areillustrative examples of embodiments and are not intended to otherwiselimit the scope of the disclosure in any way. For the sake of brevity,conventional electronics, control systems, software development andother functional aspects of the systems may not be described in detail.

The above descriptions of the disclosure are provided merely as anexample, and a person skilled in the art to which the disclosurepertains may understand that the embodiments can be easily modified intoother specific forms without changing technical ideas or essentialtechnical features of the disclosure. Therefore, it should be understoodthat the embodiments described above are provided as an example in everyaspects, and thus do not pose a limitation on the disclosure. Forexample, each component described as a single type component may beimplemented in a dispersed manner, and similarly to this, componentsdescribed as being dispersed may be implemented in an integrated manner.

The use of any and all examples, or exemplary language (e.g., “such as”)provided herein, is intended merely to better illuminate embodiments anddoes not pose a limitation on the scope of embodiments unless otherwiseclaimed.

Moreover, no item or component is essential to the practice ofembodiments unless the element is specifically described as “essential”or “critical.”

It will be understood by those skilled in the art that various changesin form and details may be made therein without departing from thespirit and scope of embodiments as defined by the appended claims.

As embodiments allows for various changes and numerous embodiments,example embodiments will be illustrated in the drawings and described indetail in the written description. However, this is not intended tolimit embodiments to particular modes of practice, and it is to beappreciated that all changes, equivalents, and substitutes that do notdepart from the spirit and technical scope of the disclosure areencompassed in embodiments. Thus, the embodiments should be consideredin a descriptive sense and not for purposes of limitation.

The scope of the disclosure is defined not by the detailed descriptionof embodiments but by the appended claims, and may be construed asencompassing all modifications or adaptions derived from the meaning andscope of the claims and their equivalent concepts.

In addition, the terms “ . . . part,” “module,” etc., described in thespecification refer to a unit for processing at least one function oroperation, which can be implemented by a hardware or a software, or acombination of a hardware and a software.

The “ . . . part” and “module” may be stored in an addressable storagemedium and also be implemented by a program executable by a processor.

For example, “ . . . part” and “module” may be implemented bycomponents, such as software components, object-oriented softwarecomponents, class components, and task components, processes, functions,attributes, procedures, sub-routines, segments of program codes,drivers, firmwares, micro-codes, circuits, data, databases, datastructures, tables, arrays, and variables.

Throughout the disclosure, the expressions “A may include at least oneof a1, a2 or a3” and “A may include at least one from among a1, a2, anda3” indicate that A may include only a1, only a2, only a3, both a1 anda2, both a1 and a3, both a2 and a3, all of a1, a2, and a3, or variationsthereof.

Such expressions are not intended to limit the elements constituting Ato be a1, a2, or a3. Accordingly, the above expressions should not beinterpreted exclusively as meaning that no other elements than a1, a2,and a3 can constitute A.

Also, the aforementioned expressions mean that A may include only a1,only a2, or only a3. The elements constituting A are not necessarilydetermined selectively in a certain group. For example, the aboveexpressions should not be construed as meaning that a1, a2, or a3selected from a group essentially including a1, a2, and a3 constitutesA.

1. An electronic device worn by a user, the electronic devicecomprising: a camera module capable of changing a photographingdirection; a motion sensor configured to obtain motion information aboutthe electronic device; an eye-tracking sensor configured to obtain gazeinformation about eyes of the user; a memory storing one or moreinstructions; and a processor configured to execute the one or moreinstructions, wherein the processor is configured to, by executing theone or more instructions: determine the photographing direction based onthe gaze information, determine photographing control information forchanging the photographing direction of the camera module to obtain animage captured without shaking, based on the motion information, controlthe camera module to change the photographing direction, based on thedetermined photographing control information, and obtain the imagecaptured by the camera module based on the changed photographingdirection.
 2. The electronic device of claim 1, wherein the processor isfurther configured to, by executing the one or more instructions,identify an eye movement state of the user based on the obtained gazeinformation, and determine the photographing control information basedon the identified eye movement state.
 3. The electronic device of claim2, wherein the processor is further configured to, by executing the oneor more instructions, calculate an eye movement acceleration based onthe gaze information, and identify the eye movement state based on aresult of comparing the calculated eye movement acceleration with apredetermined acceleration threshold value.
 4. The electronic device ofclaim 3, wherein the processor is further configured to, by executingthe one or more instructions: determine that the eyes of the user are ina state in which the eyes are not looking at a particular object whenthe calculated eye movement acceleration is greater than thepredetermined acceleration threshold value, and determine thephotographing control information based on the motion information andwithout considering the gaze information.
 5. The electronic device ofclaim 3, wherein the processor is further configured to, by executingthe one or more instructions: determine that the eyes of the user are ina state in which the eyes are looking at a particular object when thecalculated eye movement acceleration is less than or equal to thepredetermined acceleration threshold value, and determine thephotographing control information based on the gaze information and themotion information.
 6. The electronic device of claim 1, wherein themotion information comprises information about at least one from among adirection, an angle, and a speed of movement of the electronic device.7. The electronic device of claim 1, wherein the photographing controlinformation comprises information about at least one from among adirection, an angle, and a speed of rotation of the camera module. 8.The electronic device of claim 1, wherein the processor is furtherconfigured to, by executing the one or more instructions, determine thephotographing control information such that the photographing directionof the camera module is changed to an opposite direction of a movementdirection of the electronic device, based on the motion information. 9.An operating method of an electronic device worn by a user, theoperating method comprising: determining a photographing direction of acamera module based on gaze information about eyes of the user;determining photographing control information for changing thephotographing direction of the camera module to obtain an image capturedwithout shaking, based on motion information about the electronicdevice; controlling the camera module to change the photographingdirection, based on the determined photographing control information;and obtaining the image captured by the camera module based on thechanged photographing direction.
 10. The operating method of claim 9,wherein the determining of the photographing control informationcomprises: identifying an eye movement state of the user based on theobtained gaze information; and determining the photographing controlinformation based on the identified eye movement state.
 11. Theoperating method of claim 10, wherein the identifying of the eyemovement state comprises: calculating an eye movement acceleration basedon the gaze information; and identifying the eye movement state based ona result of comparing the calculated eye movement acceleration with apredetermined acceleration threshold value.
 12. The operating method ofclaim 11, wherein the identifying of the eye movement state furthercomprises determining that the eyes of the user are in a state in whichthe eyes are not looking at a particular object when the calculated eyemovement acceleration is greater than the predetermined accelerationthreshold value, and when the eyes of the user are in the state in whichthe eyes are not looking at the particular object, the determining ofthe photographing control information comprises determining thephotographing control information based on the motion information andwithout considering the gaze information.
 13. The operating method ofclaim 11, wherein the identifying of the eye movement state furthercomprises determining that the eyes of the user are in a state in whichthe eyes are looking at a particular object when the calculated eyemovement acceleration is less than or equal to the predeterminedacceleration threshold value, and the determining of the photographingcontrol information comprises determining the photographing controlinformation based on the gaze information and the motion informationwhen the eyes of the user are in the state in which the eyes are lookingat the particular object.
 14. The operating method of claim 9, whereinthe determining of the photographing control information comprisesdetermining the photographing control information such that thephotographing direction of the camera module is changed to an oppositedirection of a movement direction of the electronic device, based on themotion information.
 15. The operating method of claim 9, wherein themotion information about the electronic device comprises at least one ofa direction, an angle or a speed of movement of the electronic device.16. The operating method of claim 9, wherein the photographing controlinformation comprises at least one of a direction, an angle, or a speedof rotation of the camera module.
 17. A non-transitory computer-readablerecording medium storing a program that is executed by a processor toperform a method comprising: determining a photographing direction of acamera module based on gaze information about eyes of a user;determining photographing control information for changing thephotographing direction of the camera module to obtain an image capturedwithout shaking, based on motion information about the electronic deviceworn by the user; controlling the camera module to change thephotographing direction, based on the determined photographing controlinformation; and obtaining the image captured by the camera module basedon the changed photographing direction.